{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11.3 GAN图像生成示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3.1 DCGAN图像场景生成"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DCGAN原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[DCGAN](https://arxiv.org/abs/1511.06434)是GAN的一个变体，DCGAN就是将CNN和原始的GAN结合到一起，生成网络和鉴别网络都运用深度卷积神经网络替换了原始GAN的全连接网络。\n",
    "\n",
    "DCGAN提高了基础GAN的稳定性和生成结果质量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相较于传统的GAN网络，DCGAN在结构上主要进行了如下改变：\n",
    "\n",
    "• 取消所有pooling层。生成器中使用transposed卷积替代，判别器中使用strided卷积替代。\n",
    "\n",
    "• 使用Batch Normalization。\n",
    "\n",
    "• 移除全连接的隐层，使网络变为全卷积网络。\n",
    "\n",
    "• 在生成器上中，除了输出层使用Tanh外，其它所有层的激活函数都使用ReLU。\n",
    "\n",
    "• 在判别器中，所有层都使用LeakyReLU作为激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "生成器结构示意图如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![0](images/11-0.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文中图像生成示例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "卧室场景："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![1](images/11-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "人脸："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![2](images/11-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DCGAN图像场景生成示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DCGAN方法也可以用于遥感图像场景生成，考虑到遥感场景的复杂性，在此仅使用较为简单的农田场景作为演示示例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "农田场景示例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![3](images/11-3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DCGAN生成结果："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![4](images/11-4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[参考代码](https://colab.research.google.com/drive/1JYY_HHtVSSOLixZfLwkxiWTRdPHJCS2t#scrollTo=XkvZ4JgCHCZD)\n",
    "\n",
    "[转换过程与详细结果](https://pan.bnu.edu.cn/l/CFgIsv )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3.2 CycleGAN图像风格转换"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CycleGAN原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在用于配对图像转换的[Pix2Pix](https://openaccess.thecvf.com/content_cvpr_2017/html/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.html)方法的基础上，Zhu等人（2017）提出了用于非配对图像转换的[CycleGAN](https://openaccess.thecvf.com/content_iccv_2017/html/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.html)方法，它是传统GAN结构的变形。CycleGAN可以在没有成对训练数据的情况下将进行两个域图像之间的相互转换，目前已被应用到图像风格迁移、图像恢复、图像着色、图像去雾等领域。在图像生成领域，CycleGAN是一种能够实现非配对图像之间的风格转换的有效方法，其中网络的生成器能够生成和源域风格相近的目标域图像，也能够生成和目标域风格相近的源域图像。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![5](images/11-5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了解决配对数据不足的问题，CycleGAN引入两个镜像对称的GAN网络和循环一致性损失，消除了图像风格转换任务对配对图像的依赖。CycleGAN是由两个镜像对称的GAN网络组成的环形网络。该网络中有两个生成器G和F以及两个判别器Dx和Dy。在每个GAN结构中，生成器用于生成逼真的图像，判别器用于辨别是真实图像还是生成图像，两者之间相互博弈，最终可以利用生成器生成和实际数据分布极为相似的图像。CycleGAN模型通过最小化图像经过两次转换后和原始图像的差异来对训练过程进行约束，就能实现两个域图像风格之间的相互转换。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![6](images/11-6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中具体损失函数与目标函数定义如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "生成对抗损失："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![7](images/11-7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "循环一致性损失："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![8](images/11-8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总目标函数："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![9](images/11-9.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原文中图像风格转换示例结果："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![10](images/11-10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CycleGAN图像风格转换示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CycleGAN方法也可以用于遥感图像处理领域，由于采集的时间、地点以及成像条件不同，不同的遥感图像通常也会表现极大的风格差异。如下为使用灾害前后的拍摄的遥感图像进行风格转换的示例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "灾前图像示例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![11](images/11-11.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "灾后图像示例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![12](images/11-12.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "转换结果示例："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![13](images/11-13.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![14](images/11-14.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[参考代码](https://github.com/aitorzip/PyTorch-CycleGAN)\n",
    "\n",
    "[转换过程与详细结果](https://pan.bnu.edu.cn/l/CFgIsv )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "许多研究将这种风格差异应用到了深度神经网络模型的迁移学习研究中: [1](https://www.mdpi.com/2072-4292/13/5/984),[2](https://proceedings.mlr.press/v80/hoffman18a.html),[3](https://www.sciencedirect.com/science/article/pii/S0925231220305464),[4](https://www.sciencedirect.com/science/article/pii/S0925231220311103)。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}